Why ?

SONAR depends on https://github.com/facebookresearch/fairseq2 which introduced a ton of backward-incompatible changes in its recent 0.5 version. We update our code in order to keep up (also addressing #78).

How ?

Mostly by syncing the changes from the internal repo, including:

• Refactoring the model cards by moving tokenizer-related fields one level down
• Dealing with a lot of renamed interfaces (such as Tokenizer=>TextTokenizer)
• Using BatchLayout to describe batches
• Changing the way the model families are created (moving code from handler.py to checkpoint.py and config.py)
• Start using DependencyContainer for registering models and DependencyResolver for retrieving them

Additional changes, not directly required by fairseq2 migration:

• Getting rid of requirements.txt and requirements-dev.txt to use only pyproject.toml as a single reference
• A few extra tips in README.md

The sonar-space package version gets upgraded from 0.4.0 to 0.5.0.

Test plan

The Github CI + local integration tests (python -m pytest tests/integration_tests/; you may need to remove ~/.cache/fairseq2 if there are any problems with fairseq2 assets)